Driving assistance apparatus and driving assistance method

ABSTRACT

Provided is a method for use by a driving assistance apparatus that assists a transition from an autonomous driving mode in which a vehicle is driven under autonomous control to a manual driving mode in which the vehicle is driven by a driver. The method includes: detecting an activity by the driver; detecting conditions of the driver; and determining a take-over request method which is a method of presenting, in the vehicle, a take-over request that informs the driver that the autonomous driving mode is going to be cancelled, the determining being based on at least the detected activity by the driver and the detected conditions.

This is a continuation application of U.S. patent application Ser. No.15/607,992, filed May 30, 2017, which claims the benefit of JapanesePatent Application No. 2016-127972, filed on Jun. 28, 2016. The entiredisclosure of each of the above-identified applications, including thespecification, drawings, and claims, is incorporated herein by referencein its entirety.

BACKGROUND 1. Technical Field

The present disclosure relates to a driving assistance apparatus and adriving assistance method that assist a transition from an autonomousdriving mode in which the vehicle is driven under autonomous control toa manual driving mode in which the vehicle is driven by a driver.

2. Description of the Related Art

Research and development of autonomous driving systems have beenactively pursued in recent years. An autonomous driving system enablesdriving controls of a vehicle (such as accelerating, steering, andbraking) to be performed automatically. For autonomous driving systems,widely used are autonomy levels which are defined from the viewpoint ofthe division of roles between the driver and the system. For example,according to the definition by the National Highway Traffic and SafetyAdministration in the United States Department of Transportation, thereare five autonomy levels from Level 0 being manual driving to Level 4being fully autonomous driving.

Currently-commercialized autonomous driving systems are of Levels 0 to 2according to the above definition. Level 2 is an autonomy level at whichthe system controls more than one of the accelerating, steering, andbraking, but the driver needs to constantly monitor the drivingsituation. It is expected that Level-3 autonomous driving systems willbe commercialized in the near future. Level 3 is an autonomy level atwhich the driver does not need to monitor the driving situation whilethe system performs all of the accelerating, steering, and braking, butneeds to be ready to take over the driving when the system reaches itsfunctional limit.

The driver of a vehicle having a Level-3 autonomous driving system doesnot need to perform any driving task such as driving the vehicle ormonitoring the driving situation, as long as the system is in stableoperation. Nevertheless, the driver still needs to take over the drivingtask from the system when the system reaches its functional limit andswitches the driving mode from the autonomous driving mode to the manualdriving mode. However, it is likely that, when the driving mode isswitched from the autonomous driving mode to the manual driving mode,the driver who has been disengaged from a driving task cannotimmediately gain a sufficient situational awareness of the surroundingenvironment. Thus, from a driving safety perspective, a method forachieving an appropriate transition from autonomous driving by thesystem to manual driving by the driver is an important issue to beovercome in order for Level-3 autonomous driving systems to becommercialized. Various methods have been proposed to this end.

For example, Japanese Unexamined Patent Application Publication No.2015-230573 (hereinafter referred to as Patent Document 1) discloses avehicle driving assistance apparatus that predicts road conditions aheadof the host vehicle, determines based on the prediction whetherautonomous driving will be continuable hereafter, and when determiningthat the autonomous driving will not be continuable, gives the driver anadvance notice that the autonomous driving mode of the system is goingto be cancelled. The vehicle driving assistance apparatus of PatentDocument 1 gives the take-over request when the distance from the hostvehicle to the predicted position at which the autonomous driving willbe cancelled falls to or below a predetermined value. When the systemgives such an advance notice of a transition from autonomous driving tomanual driving, the driver can be prepared for manual drivingbeforehand, such as gaining a situational awareness of the surroundingenvironment, and make a safe transition to manual driving.

SUMMARY

One non-limiting and exemplary embodiment provides a driving assistanceapparatus and a driving assistance method that help the driver return tomanual driving in a transition from an autonomous driving mode in whichthe vehicle is driven under autonomous control to a manual driving modein which the vehicle is driven by the driver.

In one general aspect, the techniques disclosed here feature a drivingassistance apparatus that assists a transition from an autonomousdriving mode in which a vehicle is driven under autonomous control to amanual driving mode in which the vehicle is driven by a driver. Thedriving assistance apparatus comprises one or more memories; andcircuitry that, in operation, performs operations including detecting anactivity by the driver, detecting a plurality of conditions of thedriver, and determining a take-over request method which is a method ofpresenting, in the vehicle, a take-over request that informs the driverthat the autonomous driving mode is going to be cancelled, thedetermining being based on at least the detected activity by the driverand the plurality of detected conditions.

The driving assistance apparatus and the driving assistance methodaccording to the above aspect sets the take-over request methodaccording to detection results of the driver's activity and conditions.This helps the driver return to manual driving in a transition from theautonomous driving mode in which the vehicle is driven under autonomouscontrol to the manual driving mode in which the vehicle is driven by thedriver.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of a drivingassistance apparatus according to Embodiment 1;

FIG. 2 is a functional block diagram exemplifying functions of anactivity detection section in the driving assistance apparatus;

FIG. 3 is a diagram illustrating operation of the activity detectionsection in the driving assistance apparatus;

FIG. 4 is a functional block diagram exemplifying functions of awakefulness level detection part in the driving assistance apparatus;

FIG. 5 is a functional block diagram exemplifying functions of a fatiguelevel detection part in the driving assistance apparatus;

FIG. 6 is a functional block diagram exemplifying functions of anervousness level detection part in the driving assistance apparatus;

FIG. 7 is a functional block diagram exemplifying functions of aconcentration level detection part in the driving assistance apparatus;

FIG. 8 is a flowchart illustrating autonomous-driving-mode cancellationprocessing;

FIG. 9 is a functional block diagram exemplifying functions of a drivinginformation acquisition part in the driving assistance apparatus;

FIG. 10 is a diagram illustrating an activities and conditions table inthe driving assistance apparatus;

FIG. 11 is a diagram illustrating a cancellation warning methodsdatabase in the driving assistance apparatus;

FIG. 12 is a graph illustrating the relation between the level ofconcentration and the timing to start presenting a take-over request;and

FIG. 13 is a graph illustrating the relation between the level ofnervousness and the timing to start presenting a take-over request.

DETAILED DESCRIPTION

(Underlying Knowledge Forming Basis of the Present Disclosure)

As described earlier, the driver of a vehicle with a Level-3 autonomousdriving system does not need to perform any driving task such as drivingthe vehicle or monitoring the driving situation, as long as the systemis in stable operation. Freed from any driving tasks, the driver mayengage in an activity unrelated to driving in the autonomous drivingmode, such as watching TV or reading text messages on a smartphone.Herein, a “driver” is a person assumed to perform driving tasks duringthe manual driving mode. The driver may also be a person sitting in thedriver's seat or a person preregistered at the system as a driver.

When the system reaches its functional limit, the driver needs to takeover the driving tasks from the system. If the driver has been engagingin the driving-unrelated activity for a long time, it is possible thatthere is a notable decrease in the driver's consciousness of driving,such as situational awareness of the surrounding environment andperception of the conditions of the host vehicle. When the driving modeswitches to the manual driving mode with the driver in such a condition,the driver may be slow to respond to, or even miss, objects to bewatched out for while driving the vehicle, which in the worst scenariocould lead to a traffic accident.

Hence, commercialization of Level-3 autonomous driving systems requiresa method for achieving a safe and smooth transition for a driverdisengaged from a driving task to return to a state ready for manualdriving.

As described earlier, Patent Document 1 discloses a method including:predicting road conditions ahead of the host vehicle; determining basedon the prediction whether the system will be able to continue autonomousdriving; and when determining that the autonomous driving will not becontinuable, giving the driver a notice that the autonomous driving modeof the system is going to be cancelled. According to this method, thedriver is given a notice of a transition from autonomous driving tomanual driving, and therefore can get prepared for manual driving.

However, after earnest study, the inventors of the disclosure have founda problem in a conventional driving assistance apparatus, that is, theapparatus cannot check whether the driver is truly ready for manualdriving when the autonomous driving mode is cancelled.

The method of Patent Document 1, in particular, starts presenting thetake-over request once the distance from the host vehicle to a predictedposition where the autonomous driving mode is going to cancelled fallsto or below a predetermined value. Thus, depending on the driver'scondition, such as strong drowsiness, at the time of the presentation ofthe take-over request, the take-over request may not give enough timefor the driver to be ready for manual driving. It is also possible thatthe driver does not notice an autonomous driving take-over request ifengaging in an activity such as watching TV or playing a video game witha smartphone when the take-over request is presented. The inventors ofthe present disclosure have therefore found that take-over requests needto be presented to the driver based on the driver's activity andconsciousness level towards driving in the autonomous driving mode.

Japanese Unexamined Patent Application Publication No. 2015-210660(hereinafter referred to as Patent Document 2) discloses a method thatincludes: estimating a driver's activity in an autonomous driving mode;if determining that the driver is engaging in an activity unrelated todriving, selecting a certain one of information presentation devicesbased on the activity; and presenting information to prompt the driverto stop the driving-unrelated activity. The method of Patent Document 2estimates takeover time necessary for the driver to bring themselvesfrom the driving-unrelated activity to a state ready for manual driving,and arrival time after which the system cannot continue autonomousdriving, and when the takeover time is longer than the arrival time,immediately presents information to prompt the driver to stop theactivity.

This method can change, based on the driver's activity in the autonomousdriving mode, what device to use for the presentation of the informationwhich prompts the driver to stop the driving-unrelated activity. Forexample, if the driver is engaging in an activity such as watching TV orplaying a video game with a smartphone, the above method can present, onthe screen of the smartphone, information prompting the driver to stopthe activity. The inventors of the present disclosure have figured thatsuch information prompting the driver to stop an activity can beregarded as a type of information by which a take-over request ispresented.

In the method of Patent Document 2, the timing to start presenting theinformation prompting the driver to stop their driving-unrelatedactivity is determined based on the takeover time necessary for thedriver to return from the driving-unrelated activity to manual driving.This takeover time is determined for each type of activity that thedriver may engage in.

However, when the driver is, for example, playing a video game using asmartphone in the autonomous driving mode, the driver may have beenabsorbed in the game for a long time, or may have been playing the gameintermittently while occasionally monitoring the surrounding environmentof the host vehicle. The time required for the driver to fully returnfrom the game activity to manual driving should be different betweenthese two situations. The actual takeover time is likely to be longerfor the driver who has been absorbed in the game.

The conventional method does not take this into consideration and cannotensure whether the driver can truly return to a state ready for manualdriving when the autonomous driving mode is cancelled. This is theproblem that the inventors of the present disclosure have noted in theconventional method. After earnest study, the inventors of the presentdisclosure have come to the view that the takeover time necessary toreturn to manual driving needs to be set in consideration of not onlythe type of activity that the driver is engaging in, but alsophysiological and psychological conditions of the driver engaging inthat activity.

To this end, when the driving mode is going to be switched from theautonomous driving mode to the manual driving mode, a driving assistanceapparatus according to one aspect of the present disclosure changes themethod of presenting an autonomous driving take-over request to thedriver, based on the driver's activity in the autonomous driving modeand the conditions of the driver engaging in that activity. This helpsthe driver return to manual driving safely and smoothly.

An embodiment of the present disclosure is described below withreference to the drawings.

Note that the embodiment described below illustrates comprehensive orspecific examples. Numerical values, shapes, materials, constituents,the arrangements of the constituents, the connections of theconstituents, steps, the order of the steps, and the like are justexamples and are not intended to limit the present disclosure. Theconstituents in the following embodiment which are not included in theindependent claim, which provides the broadest concept, are described asoptional constituents.

Embodiment 1

A driving assistance apparatus according to Embodiment 1 is describedbelow.

1. Configuration

The configuration of a driving assistance apparatus according to thisembodiment is described below with reference to FIG. 1. FIG. 1 is ablock diagram illustrating the configuration of a driving assistanceapparatus 1 according to this embodiment.

As shown in FIG. 1, the driving assistance apparatus 1 includes a firstsensor group 2, a vehicle-mounted devices interface 20 (“interface” willbe abbreviated as “IF” hereinbelow), an activity detection section 3, adriver conditions detection section 4, and a take-over request methoddetermination unit 5. The driving assistance apparatus 1 furtherincludes a presentation controller 6, a second sensor group 7, a drivingmode selection unit 8, and a vehicle controller 80. The drivingassistance apparatus 1 is installed in a target vehicle for autonomousdriving, and assists a transition from an autonomous driving mode inwhich the vehicle is driven under autonomous control to a manual drivingmode in which the vehicle is driven by the driver.

The first sensor group 2 is a group of sensor devices that sense varioustypes of information used for monitoring the driver of the vehicle. Inthis embodiment, the first sensor group 2 includes an in-vehicle camera21, a microphone 22, a body movement sensor 23, a blood pressure sensor24, and a heartbeat sensor 25. The sensors 21 to 35 of the first sensorgroup 2 and the vehicle-mounted devices IF 20 form an example of asensor part in this embodiment.

The in-vehicle camera 21 is an example of a first camera that capturesan image of an inside of the vehicle. The in-vehicle camera 21 generatesimage data that indicates a captured image. As an example, thisembodiment assumes that the driver is in the driver's seat. Thein-vehicle camera 21 is placed inside the vehicle to be able to capturean image near the driver's seat.

The microphone 22 is placed inside the vehicle and picks up ambientsound. The microphone 22 generates data (audio data) that indicatespicked-up sound.

The body movement sensor 23 is formed by, for example, a load sensorplaced inside the backrest or seat base of the driver's seat. The bodymovement sensor 23 senses a change in the posture of a person in thedriver's seat, and generates a sensor signal indicative of the sensedresult. The body movement sensor 23 may be formed by other sensors suchas an acceleration sensor or an angular velocity sensor.

The blood pressure sensor 24 measures the blood pressure of the driverand generates a sensor signal indicative of the measurement result. Forexample, the blood pressure sensor 24 is configured as a wearable deviceand attached beforehand to the person responsible for driving in themanual driving mode.

The heartbeat sensor 25 measures the heartbeats of the driver, andgenerates sensor signals indicative of the measurement results. Theheartbeat sensor 25 may be a contact sensor attached to a part of thebody, such as an ear lobe, or may be a contactless sensor, such as acamera, that extracts a change in the color of the face corresponding toa pulse wave.

The vehicle-mounted devices IF 20 is an interface circuit (module) thatexchanges signals with various devices in the vehicle using wireless orwired communications. The vehicle-mounted devices IF 20 performscommunications in compliance with a predetermined communicationsstandard, such as USB, HDMI (registered trademark), IEEE 1395, Wi-Fi, orBluetooth (registered trademark).

For example, the vehicle-mounted devices IF 20 communicates with anautomotive navigation system or a TV system mounted in the vehicle. Inthis embodiment, the vehicle-mounted devices IF 20 also communicatesinformation with, for example, a smartphone 65 owned by the driver. Forexample, the vehicle-mounted devices IF 20 may communicate with thesmartphone 65 wirelessly with an application for communicating with thevehicle-mounted devices IF 20 installed in the smartphone 65, or have awired connection with the smartphone 65. Instead of or in addition tothe smartphone 65, other types of portable information terminals may beused, such as a handheld game console, a tablet terminal, and a notebookPC.

The activity detection section 3 detects the driver's activity based oninformation from the sensors 21 to 25 of the first sensor group 2 andthe vehicle-mounted devices IF 20. The activity detection section 3 maydetect the driver's activity periodically. A description of how theactivity detection section 3 detects an activity will be given later.

The driver conditions detection section 4 detects conditions of thedriver related to the driver's consciousness towards driving based oninformation from the sensors 21 to 25 of the first sensor group 2 andthe vehicle-mounted devices IF 20. The driver conditions detectionsection 4 may detect the conditions periodically. Hereinbelow, theconditions of the driver detected by the driver conditions detectionsection 4 may be collectively referred to as “driver conditions”. Thedriver conditions include physiological conditions and psychologicalconditions. The physiological conditions are based on humans'physiological states, such as drowsiness and fatigue, and thepsychological conditions are based on humans' psychological states, suchas nervousness and concentration.

In this embodiment, the driver conditions detection section 4 includes,as an example, a wakefulness level detection part 41, a fatigue leveldetection part 42, a nervousness level detection part 43, and aconcentration level detection part 44, which respectively detect thelevels of wakefulness, fatigue, nervousness, and concentration of thedriver. The levels of wakefulness and fatigue are each an example of ameasure of a physiological condition of the driver, and the levels ofnervousness and concentration are each an example of a measure of apsychological condition of the driver. A description of how the driverconditions detection section 4 detects the above measures will be givenlater.

For example, the activity detection section 3 and the driver conditionsdetection section 4 are implemented to form a detection unit 10together. The detection unit 10 includes a CPU that may implementpredetermined functions in cooperation with software, and therebyimplements the functions of the activity detection section 3 and thedriver conditions detection section 4. The detection unit 10 alsoincludes an internal memory configured with a ROM, a RAM, and the like,and stores, at any time, the detection results obtained by the activitydetection section 3 and the driver conditions detection section 4 in theinternal memory.

The take-over request method determination unit 5 includes a take-overrequest method selection section 50 and a storage 55. The take-overrequest method selection section 50 includes a CPU that may implementpredetermined functions in cooperation with software, and therebyimplements the functions of a take-over request start timing calculator51, a take-over request presentation device selector 52, and a take-overrequest presentation content selector 53 (which will be describedlater). Using these functions, the take-over request method selectionsection 50 selects a method of presenting, in the vehicle, a take-overrequest which informs the driver that the autonomous driving mode isgoing to be cancelled.

The storage 55 is a storage medium in which to store programs and datanecessary for implementing the functions of the take-over request methoddetermination unit 5, and is configured with a ROM, a RAM, and the like.For example, the storage 55 retains a take-over request methods databaseD1. The take-over request methods database D1 is a database in whichdriver's activities, driver's conditions, and methods of presenting atake-over request are managed in association with one another. Detailsof the functions of the take-over request method determination unit 5will be described later.

The presentation controller 6 includes a CPU that may implementpredetermined functions in cooperation with software, and controlspresentation devices 60 capable of presenting information inside thevehicle in their respective ways. The presentation devices 60 include adisplay 61 of any vehicle-mounted device such as the navigation system,a speaker 62, a driver's seat 63, and the smartphone 65. For example,the presentation controller 6 controls the actuation of, for example, avibration actuator embedded in the driver's seat 63 or mounted on theseatbelt.

In this embodiment, besides the display 61 of any vehicle-mounteddevice, the presentation controller 6 controls the smartphone 65 as oneof the display presentation devices 60. The presentation controller 6controls what image to display on the screen of the smartphone 65 or thelike by, for example, communicating with the smartphone 65 via thevehicle-mounted devices IF 20.

The second sensor group 7 is a group of sensor devices that senseinformation used to monitor the situation outside the vehicle. Thesecond sensor group 7 includes a camera 71, a radar 72, a GPS 73, and/orthe like (see FIG. 9).

The camera 71 of the second sensor group 7 is placed at a position suchas on the exterior surface of the vehicle and is an example of a secondcamera that captures an image of an environment outside of the vehicle(e.g., surrounding vehicles). The radar 72 may measure the position ofand the distance to a vehicle or an obstacle existing near the hostvehicle. The GPS 73 receives GPS information indicative of the positionof the host vehicle from GPS satellites.

The driving mode selection unit 8 includes a CPU that may implementpredetermined functions in cooperation with software, and includes adriving information acquisition part 81, a road condition predictionpart 82, an autonomous driving continuance determination part 83, anautonomous driving cancellation determination part 84, and anautonomous/manual driving switch part 85. Based on the informationobtained by the second sensor group 7, the driving mode selection unit 8selects either the autonomous driving mode in which the vehicle isdriven under autonomous control or the manual driving mode in which thevehicle is driven by the driver.

The vehicle controller 80 is configured with a CPU that may implementpredetermined functions in cooperation with software, and controls thedriving of the vehicle based on the driving mode currently selected. Forexample, in the autonomous driving mode, the vehicle controller 80 maycontrol an electronic control unit (ECU) and actuators for acceleration,steering, and braking of the vehicle, using the information obtained bythe second sensor group 7.

In the driving assistance apparatus 1 described above, each element ofthe detection unit 10, the take-over request method determination unit5, the presentation controller 6, the driving mode selection unit 8, thevehicle controller 80, and the like may be configured by dedicatedhardware, or may be implemented by execution of a software programappropriate for the element. Each element may be implemented when aprogram executer, such as a CPU or a processor, reads and executes asoftware program recorded in a recording medium, such as a hard disk ora semiconductor memory.

Some or all of the elements constituting the above-described devices maybe formed by a single systematic large scale integration (LSI) circuit.A systematic LSI is an ultra-multifunctional LSI circuit manufactured byintegration of multiple components on one chip. Specifically, asystematic LSI circuit is a computer system including a microprocessor,a ROM, a RAM, and the like. Computer programs are stored in the RAM. Thesystematic LSI circuit achieves its functions when the microprocessoroperates according to the computer programs.

Further, some or all of the elements constituting the above-describeddevices may be formed by IC cards or modules capable of being attachedto and detached from the devices. The IC cards or modules are each acomputer system including a microprocessor, a ROM, a RAM, and the like.The IC cards or modules may include the above-describedultra-multifunctional LSI circuit. The IC cards or modules achieve theirfunctions when the microprocessor operates according to computerprograms. The IC cards or modules may be tamper-resistant.

2. Operations

Descriptions are given below of operations of the driving assistanceapparatus 1 according to this embodiment, which is configured asdescribed above.

2-1 Outline of the Operations

An outline of the operations of the driving assistance apparatus 1according to this embodiment is described with reference to FIG. 1.Using the activity detection section 3 and the driver conditionsdetection section 4, the driving assistance apparatus 1 of thisembodiment keeps monitoring an activity by the driver and conditions ofthe driver engaging in the activity while the vehicle is driving. In theautonomous driving mode, the driving mode selection unit 8 constantlydetermines, based on the road conditions and the like, whetherautonomous driving will be continuable for a predetermined period oftime (e.g., one hour) or longer from the current time.

If it is determined by the above determination processing thatautonomous driving will not be continuable for the predetermined periodof time or longer, the take-over request method determination unit 5determines a method of presenting a notice of the cancellation of theautonomous driving mode by referring to the activity by the driverdetected at that time and a detection result of a particular drivercondition corresponding to that activity. Then, the presentationcontroller 6 presents the take-over request according to the determinedmethod, including the timing for starting the presentation, apresentation device to use for the presentation, and the content of thepresentation. The “particular driver condition” is a condition (e.g., aconcentration level) deemed necessary in making a determination whetherthe driver can really return from the driving-unrelated activity tomanual driving, and is predetermined for each of activities of thedriver that may be detected.

Specifically, if the driver is engaging in a certain activity such aswatching TV or playing a video game in the autonomous driving mode, thetake-over request method determination unit 5 changes the timing tostart the presentation of a take-over request, the device to use for thepresentation (such as one that outputs sound or one that displays ascreen), and the content of the presentation, based not only on thelevel of wakefulness of the driver engaging in the activity, but also onthe level of concentration on the activity. For instance, if the driveris watching TV drowsily or is absorbed in playing a video game andextremely concentrated on the screen, it is likely that the driver needsmore time to be in a state capable of manual driving than usual. Thus,in this embodiment, the take-over request method determination unit 5makes the timing to start presenting the take-over request earlier thanusual and highlights the take-over request more than usual by selectingsuch a presentation device and content.

As described above, the driving assistance apparatus 1 according to thisembodiment changes the method of presenting a notice of autonomousdriving cancellation according to the driver's activity and conditions,helping the driver who has been disengaged from driving in theautonomous driving mode return safely and smoothly to a state capable ofmanual driving. The following provides detailed descriptions about theoperations of the driving assistance apparatus 1 according to thisembodiment.

2-2. How the Driver's Activity is Detected

With reference to FIGS. 2 and 3, a description is given of how theactivity detection section 3 detects the driver's activity. FIG. 2 is afunctional block exemplifying functions of the activity detectionsection 3 in the driving assistance apparatus 1. FIG. 3 is a diagramillustrating operation of the activity detection section 3.

As shown in FIG. 2, the activity detection section 3 includes anactivity estimator 300 that detects an activity in which the driver ispresumably engaging. The activity detection section 3 receives sensorsignals from the sensors 21, 22, and 23 of the first sensor group 2 andan output signal from the vehicle-mounted devices IF 20, andsequentially performs detection processing for recognizing the activityin which the driver is engaging while the vehicle is driving. Based onthe results of the detection processing, the activity estimator 300detects an activity by the driver among various kinds of activities,such as texting, playing a video game, watching TV, and so on. Examplesof the activities are listed in FIG. 3.

When a device communicatively connected to the vehicle-mounted devicesIF 20, such as an automotive navigation system 61 a or the smartphone65, is operated, the vehicle-mounted devices IF 20 receives operationinformation including the device being operated and the content of theoperation, and generates an output signal indicating the operationinformation.

As shown in FIG. 2, the activity detection section 3 includes anoperated-device detector 301 and an operation content detector 302.Based on the output signal from the vehicle-mounted devices IF 20, theoperated-device detector 301 detects the device indicated by theoperation information included in the output signal, and the operationcontent detector 302 detects the content of the operation indicated bythe operation information.

For instance, if the driver is playing a video game using the smartphone65, the operated-device detector 301 of the activity detection section 3detects the smartphone 65 as the device being operated, and theoperation content detector 302 of the activity detection section 3detects that the content of the operation is playing a video game. Thus,the activity estimator 300 estimates that the driver's activity is“playing a video game with a smartphone”.

As shown in FIG. 2, the activity detection section 3 further includes anupper body image detector 303, a body motion recognizer 304, aheld-object recognizer 305, a facial image extractor 306, and aviewed-object detector 307. The activity detection section 3 receivesimage data indicating an image captured by the in-vehicle camera 21, andperforms the following image recognition processing using these parts ofthe activity detection section 3 based on the image captured by thein-vehicle camera 21.

Based on the image data from the in-vehicle camera 21, the upper bodyimage detector 303 detects an image of the upper body of the driver inthe captured image. For example, with image data on an image capturedwhen no one is in the driver's seat being stored in the internal memoryin advance, the upper body image detector 303 reads the stored imagedata, compares it with image data on an image which has just beencaptured, and detects an upper body image based on the different betweenthese pieces of image data.

The body motion recognizer 304 recognizes a body motion of the driverengaging in a certain activity based on the upper body image detected.Specifically, for example, the body motion recognizer 304 may detect thepositions and postures of the arms and hands based on the contour of theupper body image. Body motions of the driver include touching the screenof a device with a finger, pushing a button on a device, moving a handand/or an arm to hold an object, and moving the mouth to eat or speak.

When the body motion of holding an object has been detected, theheld-object recognizer 305 performs image processing to recognize theobject being held. For example, the held-object recognizer 305 extractsan image of the object being held near the hand from the upper bodyimage, compares it with each piece of image data stored in advance asheld-object candidates, such as a smartphone, a game console, and abook, and determines the similarities between the extracted image andthe stored image data.

The facial image extractor 306 extracts a facial image from the detectedupper body image based for example on a feature amount related to humanfaces. Alternatively, the facial image extractor 306 may extract afacial image from the image data on the entire captured image.

The viewed-object detector 307 detects the object at which the driver islooking, based on the orientation of the face detected in the extractedfacial image or from the direction of the line of sight estimated fromimages of the eyes extracted from the facial image. For example, theviewed-object detector 307 detects an object towards which the face orthe eyes are directed, by analyzing the entire captured image orestimating the arrangement of devices in the vehicle based on theposition of the in-vehicle camera 21.

By the image recognition processing described above, the activityestimator 300 can detect an activity, such as operating a device,watching content on a device (these devices do not have to becommunicatively connected to the vehicle-mounted devices IF 20), readinga book, a document, or the like, and eating (see FIG. 3).

As shown in FIG. 2, the activity detection section 3 further includes avoice detector 308, a speaker number detector 309, an environmentalsound detector 310, and an environmental sound recognizer 311. Theactivity detection section 3 receives data on sound picked up by themicrophone 22, and performs voice recognition processing andenvironmental sound recognition processing using these parts as follows.

The voice detector 308 detects human voice by, for example, extracting afeature amount unique to a human voice from the data on sound picked upby the microphone 22. The speaker number detector 309 detects the numberof speakers based for example on the individual differences among thefeature amounts of the voices detected. Thereby, when a human voice isdetected, the activity detection section 3 can presume that the driveractivity is, as illustrated in FIG. 3, talking on the phone if there isone speaker, and talking with a passenger or passengers if there is morethan one speaker.

The environmental sound detector 310 detects an environmental sound inthe data on the sound picked up by the microphone 22, the environmentalsound being determined as not being a human voice based on predeterminedfeature amounts and the like. The environmental sound recognizer 311performs processing for identifying the environmental sound detected.For example, the environmental sound recognizer 311 compares thedetected environmental sound with each piece of environmental sound datapre-stored as environmental sound candidates, such as music, radiosound, chewing sound, or sound generated by operation of a device suchas a shaver.

As shown in FIG. 2, the activity detection section 3 further includes abody movement information detector 313. The body movement informationdetector 313 detects body movement information on a body movement of thedriver based on a sensor signal from the body movement sensor 23. Thebody movement information detector 313 thus detects a body movement thatthe driver makes to change their position during a certain one ofvarious activities, and thereby improves the accuracy of the body motiondetected to estimate the driver's activity (see FIG. 3).

2-3. How Driver's Conditions are Detected

Hereinbelow, a description is given of how the driver conditionsdetection section 4 detects driver conditions.

2-3-1. Methods for Detecting the Level of Wakefulness

With reference to FIG. 4, a description is given of how the wakefulnesslevel detection part 41 of the driver conditions detection section 4detects the level of wakefulness. FIG. 4 is a functional block diagramexemplifying functions of the wakefulness level detection part 41 in thedriving assistance apparatus 1.

In this embodiment, the level of wakefulness is a measure of how awakethe driver is, as opposed to how drowsy they are. Hence, when drowsy,the driver has a low level of wakefulness.

The wakefulness level detection part 41 detects the level of wakefulnessof the driver by, for example, analyzing an image captured by thein-vehicle camera 21. As shown in FIG. 4, the wakefulness leveldetection part 41 includes a facial image extractor 411, an eye openingdegree detector 412, an eye blink detector 413, a head position detector414, and a wakefulness level estimator 410.

The wakefulness level detection part 41 performs image analysis usingits parts as follows. First, the facial image extractor 411 acquiresimage data from the in-vehicle camera 21 at any time, and extracts afacial image of the driver from the captured image. Next, the eyeopening degree detector 412 detects the eye opening degree indicatinghow much the eyelids are open in the extracted facial image. It islikely that the drowsier the driver is, the more the eyelids are closed,making the eye opening degree smaller. Thus, the wakefulness levelestimator 410 calculates an estimated level of wakefulness based on theeye opening degree detected: the smaller the eye opening degree, thelower the level of wakefulness.

In addition, when the driver is drowsy, the period of time the eyelidsare closed in one blink is likely to be long, making the number ofblinks fewer. Thus, the eye blink detector 413 detects how long theeyelids are closed and how many times the eyes blink based on facialimages captured over a predetermined period of time (e.g., one minute).The wakefulness level estimator 410 estimates the level of wakefulnessbased on the eye-closed time and the number of eye blinks detected: thelonger the eye-closed time and/or the smaller the number of eye blinks,the lower the level of wakefulness.

When the driver is drowsy, the position of the head may be unstable.Thus, the head position detector 414 detects the position of the head ofthe driver based on the position of the facial image extracted from theimage captured by the in-vehicle camera 21. The wakefulness levelestimator 410 estimates the level of wakefulness based on detectionresults of the position of the head over a predetermined period of time(e.g., one minute): the larger the rate of change in the position of thehead, the lower the level of wakefulness.

In addition to or instead of the above analysis of captured images, thewakefulness level detection part 41 may detect the level of wakefulnessby analyzing voice in the vehicle. When the driver is asleep, sleepingsound such as a snore may be observed. In view of this, the wakefulnesslevel detection part 41 includes a breathing sound detector 415. Thebreathing sound detector 415 detects breathing sound of the driver fromdata on sound picked up by the microphone 22, and determines whether thedetected breathing sound contains sleeping sound such as a snore. Thewakefulness level estimator 410 lowers the estimated level ofwakefulness every time it determines that the detected breathing soundcontains sleeping sound.

The wakefulness level detection part 41 may detect the level ofwakefulness based on a measurement result from the heartbeat sensor 25.It is known that in heart rate variability (HRV) containing a highfrequency (HF) component (e.g., 0.15 to 0.40 Hz) and a low frequency(LF) component (e.g., 0.04 Hz to 0.15 Hz), the HF component, which isattributable to a parasympathetic activity, is elevated when humans aredrowsy.

In view of this, as shown in FIG. 4, the wakefulness level detectionpart 41 includes a heart rate variation detector 416 and an LF/HFdetector 417. The heart rate variation detector 416 detects heart ratevariation based on sensor signals from the heartbeat sensor 25. TheLF/HF detector 417 detects the ratio of the LF component to the HFcomponent (LF/HF) in the heart rate variation detected. The wakefulnesslevel estimator 410 estimates the level of wakefulness based on theLF/HF ratio detected: the smaller the LF/HF ratio, the lower the levelof wakefulness.

As shown in FIG. 4, the wakefulness level detection part 41 may includea body movement information detector 418. When the driver moves often,the driver is likely to be awake. In view of this, the body movementinformation detector 418 detects, based on sensor signals from the bodymovement sensor 23, body movement information indicating that there hasbeen a body movement of the driver. The wakefulness level estimator 410estimates the level of wakefulness based on body movements detectedbased on the body movement information: the more the body movements, thehigher the level of wakefulness.

As shown in FIG. 4, the wakefulness level detection part 41 may includean operation frequency detector 419. When a device is operatedfrequently, the driver is likely to be awake. Thus, the operationfrequency detector 419 detects an operation frequency based on operationinformation included in output signals from the vehicle-mounted devicesIF 20 over the past predetermined time (e.g., one minute). The operationfrequency indicates the number of times a device is operated within apredetermined period of time. The wakefulness level estimator 410estimates the level of wakefulness based on the operation frequencydetected: the higher the operation frequency, the higher the level ofwakefulness.

In the examples described above, the wakefulness level estimator 410estimates the level of wakefulness according to a result obtained by anyof the eye opening degree detector 412, the eye blink detector 413, thehead position detector 414, the breathing sound detector 415, the LF/HFdetector 417, the operation frequency detector 419, and the bodymovement information detector 418. However, the present disclosure isnot limited to such examples, and the wakefulness level estimator 410may estimate the level of wakefulness by considering all the resultsfrom the detectors comprehensively.

2-3-2. Methods for Detecting the Level of Fatigue

With reference to FIG. 5, a description is given of how the fatiguelevel detection part 42 of the driver conditions detection section 4detects the level of fatigue. FIG. 5 is a functional block diagramexemplifying functions of the fatigue level detection part 42 in thedriving assistance apparatus 1.

In this embodiment, the level of fatigue is a measure of how tired thedriver is physically.

For example, the fatigue level detection part 42 detects the level offatigue based on measurement results from the heartbeat sensor 25. Asshown in FIG. 5, the fatigue level detection part 42 includes a heartrate detector 421, a heart rate variation detector 422, an LF/HFdetector 423, and a fatigue level estimator 420.

The heart rate detector 421 detects the heart rate (HR) of the driverbased on sensor signals from the heartbeat sensor 25. When the driver istired, their heart rate may depart from the average heart rate at rest.In view of this, the fatigue level estimator 420 calculates an estimatedlevel of fatigue based on the difference between the detected heart rateand the preset standard heart rate: the larger the difference, thehigher the level of fatigue.

In addition, the heart rate variation detector 422 and the LF/HFdetector 423 of the fatigue level detection part 42 detect a heart ratevariation and an LF/HF ratio, respectively, as those of the wakefulnesslevel detection part 41 do. In the heart rate variation, the LFcomponent, which is attributable to a sympathetic activity, is thoughtto be elevated when the driver is tired. In view of this, the fatiguelevel estimator 420 estimates the level of fatigue based on the LF/HFratio detected: the larger the LF/HF ratio, the higher the level offatigue.

Further, as shown in FIG. 5, the fatigue level detection part 42 mayinclude a body movement information detector 424 that detects bodymovement information as that of the wakefulness level detection part 41does. When the driver is tired, it is likely that the driver frequentlychanges their posture to adjust themselves or the like, and thereforemoves more often than usual. In view of this, the fatigue levelestimator 420 estimates the level of fatigue based on the detected bodymovement information: the more the driver has been moving, the higherthe level of fatigue.

In addition, as shown in FIG. 5, the fatigue level detection part 42 mayinclude an operation frequency detector 425 and an incorrect operationdetector 426. As the operation frequency detector 419 of the wakefulnesslevel detection part 41 does, the operation frequency detector 425 ofthe fatigue level detection part 42 detects operation frequency based onoperation information from the vehicle-mounted devices IF 20. Theincorrect operation detector 426 detects incorrect operation frequencybased on operation content included in the operation information fromthe vehicle-mounted devices IF 20. The incorrect operation frequency isthe number of incorrect operations performed within a predeterminedperiod of time.

In is assumed in this embodiment that the driver gets tired fromperforming a driving-unrelated activity in the autonomous driving mode.For example, if the driver feels tired when operating a smartphone, itis likely that the operation frequency decreases and the number ofincorrect operations increases. In view of this, the fatigue levelestimator 420 estimates the level of fatigue based on the operationfrequency and the incorrect operation frequency detected: the lower theoperation frequency and/or the higher the detected incorrect operationfrequency, the higher the level of fatigue.

In the examples described above, the fatigue level estimator 420estimates the level of fatigue according to a result obtained by any ofthe heart rate detector 421, the heart rate variation detector 422, theLF/HF detector 423, the body movement information detector 424, theoperation frequency detector 425, and the incorrect operation detector426. However, the present disclosure is not limited to such examples,and the fatigue level estimator 420 may estimate the level of fatigue byconsidering all the results from the detectors comprehensively.

2-3-3. Methods for Detecting the Level of Nervousness

With reference to FIG. 6, a description is given of how the nervousnesslevel detection part 43 of the driver conditions detection section 4detects the level of nervousness. FIG. 6 is a functional block diagramexemplifying functions of the nervousness level detection part 43 in thedriving assistance apparatus 1.

In this embodiment, the level of nervousness is a measure of how nervous(or excited) the driver is.

For example, the nervousness level detection part 43 detects the levelof nervousness based on a measurement result from the blood pressuresensor 24. As shown in FIG. 6, the nervousness level detection part 43includes a blood pressure detector 431 and a nervousness level estimator430.

The blood pressure detector 431 detects the blood pressure of the driverbased on a sensor signal from the blood pressure sensor 24. Humans tendto have a higher blood pressure when they are nervous. In view of this,the nervousness level estimator 430 calculates an estimated level ofnervousness based on the blood pressure detected: the higher the bloodpressure, the higher the level of nervousness.

As shown in FIG. 6, the nervousness level detection part 43 may includea heart rate detector 432, a heart rate variation detector 433, and anLF/HF detector 434 that detect, respectively, a heart rate, a heart ratevariation, and an LF/HF ratio as those in the fatigue level detectionpart 42 do.

Humans tend to have a higher heart rate when they are nervous. In viewof this, the nervousness level estimator 430 estimates the level ofnervousness based on the heart rate detected: the higher the heart rate,the higher the level of nervousness. Moreover, humans tend to have aheart rate variation with an elevated LF component when they arenervous. In view of this, the nervousness level estimator 430 estimatesthe level of nervousness based on the LF/HF ratio detected: the largerthe LF/HF ratio, the higher the level of nervousness.

In addition, as shown in FIG. 6, the nervousness level detection part 43may include an incorrect operation detector 435 that detects thefrequency of incorrect operations as that of the fatigue level detectionpart 42 does. When nervous while operating a smartphone or the like, thedriver tends to perform incorrect operations more than usual. In view ofthis, the nervousness level estimator 430 estimates the level ofnervousness based on the incorrect operation frequency detected: thegreater the incorrect operation frequency, the higher the level ofnervousness.

In the examples described above, the nervousness level estimator 430estimates the level of nervousness according to a result obtained by anyof the blood pressure detector 431, the heart rate detector 432, theheart rate variation detector 433, the LF/HF detector 434, and theincorrect operation detector 435. However, the present disclosure is notlimited to such examples, and the nervousness level estimator 430 mayestimate the level of nervousness by considering all the results fromthe detectors comprehensively.

2-3-4. Methods for Detecting the Level of Concentration

With reference to FIG. 7, a description is given of how theconcentration level detection part 44 of the driver conditions detectionsection 4 detects the level of concentration. FIG. 7 is a functionalblock diagram exemplifying functions of the concentration leveldetection part 44 in the driving assistance apparatus 1.

In this embodiment, the level of concentration is a measure of how muchthe driver is concentrated on the activity in which they are engaging.

For example, the concentration level detection part 44 detects the levelof concentration by analyzing images captured by the in-vehicle camera21. As shown in FIG. 7, the concentration level detection part 44includes a facial image extractor 441, an eye movement detector 442, anda concentration level estimator 440. The facial image extractor 441 ofthe concentration level detection part 44 extracts a facial image asthat of the wakefulness level detection part 41 does.

The eye movement detector 442 identifies the positions of the eyes inthe facial images extracted over a predetermined period of time (e.g.,one minute) and detects the movement of the eyes, that is, an eyemovement rate. When the driver is highly concentrated on an activity inwhich they are engaging, they tend to fix their eyes on the objectinvolved in the activity (such as the screen of a portable gameconsole), making fewer eye movements. In view of this, the concentrationlevel estimator 440 calculates an estimated level of concentration basedon the eye movement rate detected: the lower the eye movement rate, thehigher the level of concentration.

As shown in FIG. 7, in addition to or instead of the eye movementdetector 442, the concentration level detection part 44 may include aneye opening degree detector 443 that detects how much the eyes are openas that of the wakefulness level detection part 41 does, and/or an eyeblink detector 444 that detects the number of eye blinks as that of thewakefulness level detection part 41 does. When the driver isconcentrated, it is likely that they open their eyes wider and blinkless. In view of this, the concentration level estimator 440 estimatesthe level of concentration based on the eye opening degree detectedand/or the number of eye blinks detected: the larger the eye openingdegree and/or the smaller the number of eye blinks, the higher the levelof concentration.

As shown in FIG. 7, the concentration level detection part 44 mayfurther include an operation frequency detector 445 that detects theoperation frequency as that of the fatigue level detection part 42 does,and/or an incorrect operation frequency detector 446 that detects theincorrect operation frequency as that of the fatigue level detectionpart 42 does. For example, when the driver is playing a video game witha portable game console and is highly concentrated, it is likely thatthe driver performs more operations on the console than usual andperform fewer incorrect operations than usual. In view of this, theconcentration level estimator 440 estimates the level of concentrationbased on the operation frequency detected and/or the incorrect operationfrequency detected: the higher the operation frequency and/or the lowerthe incorrect operation frequency, the higher the level ofconcentration.

As shown in FIG. 7, the concentration level detection part 44 mayfurther include a heart rate detector 447 that detects heart rate asthat of the fatigue level detection part 42 does. When the driver ishighly concentrated, it is likely that the driver has a higher heartrate than usual, irrespective of the kind of activity in which they areengaging. In view of this, the concentration level estimator 440estimates the level of concentration based on the heart rate detected:the higher the heart rate, the higher the level of concentration.

In the examples described above, the concentration level estimator 440estimates the level of concentration according to a result obtained byany of the facial image extractor 441, the eye movement detector 442,the eye opening degree detector 443, the eye blink detector 444, theoperation frequency detector 445, the incorrect operation frequencydetector 446, and the heart rate detector 447. However, the presentdisclosure is not limited to such examples, and the concentration levelestimator 440 may estimate the level of concentration by considering allthe results from the detectors comprehensively.

2-4. Processing for Cancelling the Autonomous Driving Mode

With reference to FIGS. 1 and 8, a description is given of cancellationprocessing executed for cancellation of the autonomous driving mode.FIG. 8 is a flowchart illustrating how the driving assistance apparatus1 performs the cancellation processing to cancel the autonomous drivingmode.

The cancellation processing as illustrated in the flowchart of FIG. 8 isperformed by the driving assistance apparatus 1 in the autonomousdriving mode. During this cancellation processing, the activitydetection section 3 and the driver conditions detection section 4iterate their detection operations described above.

First, in the driving mode selection unit 8 of the driving assistanceapparatus 1 (see FIG. 1), the road condition prediction part 82 predictsroad conditions based on driving information from the drivinginformation acquisition part 81 (S1). For example, if the host vehicleis travelling on an expressway, the road condition prediction part 82predicts the time it will take to arrive at the exit of the expressway.The driving information indicates the driving conditions of the hostvehicle and the surrounding environment of the host vehicle. Adescription will be given later of how the driving information isacquired.

Next, based on the prediction results of the road conditions and thedriving information, the autonomous driving continuance determinationpart 83 determines whether the autonomous driving mode will becontinuable for a predetermined period of time or longer from thecurrent time (S2). This predetermined period of time is set with amargin for advancing the timing to start presenting a take-over requestto the maximum degree, or in other words, the time considered enough tobring the driver who has been disengaged from driving back to a statecapable of manual driving, well in advance of the cancellation of theautonomous driving mode. An example of the predetermined period of timeis ten minutes.

If it is determined that the autonomous driving mode will be continuable(Yes in S2), the driving mode selection unit 8 returns to Step S1 anditerates the processing at a predetermined interval (e.g., one minute).

If it is determined that the autonomous driving will not be continuable(No in S2), the take-over request method selection section 50 of thetake-over request method determination unit 5 acquires information onthe activity detected by the activity detection section 3 andinformation on the driver's conditions detected by the driver conditionsdetection section 4, these pieces of information being ones detected atthe time of the determination. Based on the thus-acquired detectionresults of the driver's activity and conditions, the take-over requestmethod determination unit 5 determines a take-over request method (S3).

The take-over request method specifies how a take-over request ispresented inside the vehicle. In Step S3, the take-over request starttiming calculator 51 of the take-over request method selection section50 calculates the timing to start presenting a take-over request, andsets the timing in, for example, the presentation controller 6. Thetake-over request presentation device selector 52 selects, from thepresentation devices 60, a presentation device to use to present thetake-over request, and sets the presentation device in the presentationcontroller 6. The take-over request presentation content selector 53selects the content to present in the take-over request and sets thepresentation content in the presentation controller 6. A detaileddescription will be given of processing for determining the take-overrequest method.

Next, the presentation controller 6 determines whether the current timehas reached the start timing set by the take-over request start timingcalculator 51 (S4). The presentation controller 6 iterates this stepuntil the current time reaches the set start timing.

When the current time reaches the set start timing (Yes in S4), thepresentation controller 6 presents the take-over request according tothe take-over request method determined by the take-over request methoddetermination unit 5 (S5). Specifically, the presentation controller 6controls the presentation of the take-over request so that the take-overrequest may be presented with the selected content using the selectedpresentation device.

Next, based on the driver's activity and conditions detected after thepresentation of the take-over request, the autonomous drivingcancellation determination part 84 of the driving mode selection unit 8determines whether the driver is ready for manual driving (S6). Theautonomous driving cancellation determination part 84 performs thisprocessing by determining, for example, whether the driver has stoppedthe driving-unrelated activity or whether the driver is looking ahead ofthe vehicle. This determination may also be made using detection resultsof physiological/psychological conditions, such as the driver'swakefulness level (by checking whether the wakefulness level is not toolow), when appropriate.

If it is determined that the driver is not ready for manual driving (Noin S6), the take-over request method determination unit 5 determines atake-over request method anew (S3). In this case, the take-over requestmethod is re-set considering the fact that the take-over request isgoing to be presented a second time.

If it is determined that the driver is ready for manual driving (Yes inS6), the autonomous/manual driving switch part 85 controls the vehiclecontroller 80 so that the vehicle controller 80 will cancel theautonomous driving mode and switch the driving mode to the manualdriving mode (S7). Then, the vehicle controller 80 cancels theautonomous driving control of the vehicle.

With this, the driving mode of the vehicle is switched to the manualdriving mode, and the driving assistance apparatus 1 ends the processingillustrated in the flowchart of FIG. 8.

Before a transition from the autonomous driving mode to the manualdriving mode, the above-described cancellation processing changes thetake-over request method in accordance with an activity in which thedriver is engaging in the autonomous driving mode as well asphysiological/psychological conditions of the driver engaging in thatactivity. This enables the driver to safely and smoothly return tomanual driving.

In the example described above, after the determination in Step S3, thepresentation controller 6 iterates the determination step S4 until thecurrent time reaches the set start timing. However, the presentdisclosure is not limited to this. For example, it is possible that thedriver's activity and conditions may change between the current time andthe set start timing. For this reason, if for example the determinationresult of Step S4 is “No”, the take-over request method determinationunit 5 may check new detection results from the activity detectionsection 3 and the driver conditions detection section 4. If there is anychange in the driver's activity or conditions, the take-over requestmethod determination unit 5 may update the take-over request methodaccording to the change.

Moreover, in the example described above, when the autonomous drivingcancellation determination part 84 determines that the driver is notready for manual driving (No in S6), the take-over request methoddetermination unit 5 determines the take-over request method anew (S3).If the determination in Step S6 turns out to be “No” a predeterminednumber of times or more, the flowchart may be ended. Then, the vehiclecontroller 80 may control the vehicle so that, for example, the vehiclewill be parked in a safe place.

2-4-1. How the Driving Information is Acquired

With reference to FIG. 9, a description is given of how the drivinginformation in Step S1 of the flowchart in FIG. 8 is acquired. FIG. 9 isa functional block diagram exemplifying functions of the drivinginformation acquisition part 81.

As shown in FIG. 9, the driving information acquisition part 81 includesa vehicle travel position calculator 811, a map data storage 812, avehicle travel information acquirer 813, a surrounding environmentinformation acquirer 814, and a traffic information acquirer 815. Thesecond sensor group 7 includes the camera 71, the radar 72, and the GPS73. It is assumed that the vehicle is equipped with a wirelesscommunication unit 11 as a communication module capable of wirelesslycommunicating with a network such as the Internet.

The driving information acquisition part 81 acquires GPS informationfrom the GPS 73 of the second sensor group 7. The map data storage 812has map data stored therein in advance. Based on the GPS informationthus acquired, the vehicle travel position calculator 811 calculates thetravelling position of the host vehicle on the map data stored in themap data storage 812.

The vehicle travel information acquirer 813 acquires vehicle travelinformation indicating travelling states, such as a travelling speed, ofthe host vehicle whose driving is controlled by the vehicle controller80.

If, for example, the host vehicle is travelling on an expressway in theautonomous driving mode, in Step S1 of the flowchart in FIG. 8, the roadcondition prediction part 82 estimates, based on tinformation of thevehicle position, the map data, and the vehicle travel information, thetime it will take for the host vehicle to arrive at the exit of theexpressway. Then, in Step S2 of the flowchart in FIG. 8, the autonomousdriving continuance determination part 83 compares the estimated timeand the predetermined period of time which has been set in advance (see2-4), and proceeds to “No” when the estimated time falls below thepredetermined time.

The surrounding environment information acquirer 814 acquires image dataon captured images of the environment outside the host vehicle from thecamera 71. The surrounding environment information acquirer 814 may alsoacquire measurement information from the radar 72. The image data fromthe camera 71 and the measurement information from the radar 72 are eachan example of surrounding environment information including informationon vehicles surrounding the host vehicle.

The traffic information acquirer 815 acquires traffic information fromthe wireless communication unit 11. The traffic information indicatesreal-time traffic conditions, such as the location and size of trafficcongestion, which are available on the Internet or the like.

Based on information such as the surrounding environment informationacquired by the surrounding environment information acquirer 814 and thetraffic information acquired by the traffic information acquirer 815,the road condition prediction part 82 predicts, for example, the trafficconditions surrounding the host vehicle within the aforementionedpredetermined period of time. For instance, the road conditionprediction part 82 predicts whether the host vehicle would pass thecongested area within the predetermined time. Based on a result of theprediction by the road condition prediction part 82, the autonomousdriving continuance determination part 83 determines whether theautonomous driving mode will be continuable for the predetermined periodof time or longer from the current time, or in other words, whether theautonomous driving system will face its functional limit.

2-4-2. Processing for Determining the Take-Over Request Method

With reference to FIG. 10 to 13, a description is given of processingfor determining the take-over request method in Step S3 of the flowchartin FIG. 8.

FIG. 10 shows an example of an activities and conditions table D2. InStep S3 of the flowchart in FIG. 8, the take-over request methoddetermination unit 5 refers to the activities and conditions table D2and selects, from various measures indicative of driver conditions, ameasure used to determine the take-over request method. The activitiesand conditions table D2 is a data table in which each driver activity isrecorded in association with measures of driver conditions which shouldbe considered for the driver activity.

In the activities and conditions table D2 exemplified in FIG. 10,detection target activities are each associated with wakefulness level,fatigue level, nervousness level, and concentration level. The detectiontarget activities include texting, using SNS, browsing the Web, playinga video game, watching TV, reading, listening to music, listening to theradio, conversing, talking on the phone, having a meal, and performingpersonal maintenance (such as shaving or putting on a makeup). In StepS3 of the flowchart in FIG. 8, the take-over request method selectionsection 50 selects measures that are assigned “O” in the activities andconditions table D2 for the activity detected by the activity detectionsection 3.

In the activities and conditions table D2 exemplified in FIG. 10,concentration level is assigned “O” for activities such as texting,using SNS, browsing the Web, playing a video game, watching TV, having ameal, and performing personal maintenance. This is because when thedriver is concentrated on such activities, it is likely that the driverfixes their eyes to a particular object (such as the game screen ifplaying a video game) and pays almost no attention to, for example, theenvironment surrounding the host vehicle. This means it may take alonger time for the driver to return to manual driving.

No matter what the activity, even for an activity such as talking on thephone, if the driver is in a psychological state where they feelnervous, it is likely that it would take time for the driver to returnto manual driving. In view of this, nervousness level is assigned “0”for all the activities exemplified in FIG. 10.

Some activities may make the driver tired or drowsy. Thus, the fatiguelevel and the wakefulness level are assigned “0” for such particularactivities exemplified in FIG. 10.

From the driver conditions detection section 4, the take-over requestmethod determination unit 5 acquires estimated levels of the respectivemeasures selected as described above based on the activity detected bythe activity detection section 3, and determines a take-over requestmethod by referring to the take-over request methods database D1.

FIG. 11 shows an example of the take-over request methods database D1.The take-over request methods database D1 manages driver's activities,driver's conditions, and take-over request methods in association withone another. FIG. 11 shows an example of the take-over request methodsdatabase D1 where the driver's activities are categorized into twogroups: one with texting, using SNS, browsing the Web, playing a videogame, and watching TV, and one with reading. In this case, the take-overrequest method determination unit 5 determines a take-over requestmethod based on the levels of wakefulness, fatigue, nervousness, andconcentration (see FIG. 10).

As an example, the take-over request methods database D1 exemplified inFIG. 11 has each driver condition ranked “low”, “medium”, or “high”, andmanages take-over request methods corresponding to these three drivercondition ranks. Specifically, the take-over request methoddetermination unit 5 ranks each of the estimated levels acquired fromthe driver conditions detection section 4 “low”, “medium”, or “high” bycomparing the estimated level with two thresholds levels.

In the example depicted in FIG. 11, if the levels of wakefulness,fatigue, nervousness, and concentration are ranked “middle”, itindicates that the driver's condition is normal. In this case, thetake-over request presentation content selector 53 of the take-overrequest method determination unit 5 sets the presentation content of thetake-over request to be “notify the driver of switch to manual driving”.The take-over request presentation device selector 52 selects a displaysystem and a speaker system as presentation devices. Thereby, as shownin FIG. 11, as a take-over request, this notification content isdisplayed on a certain screen such as the automotive navigation system61 a, and outputted as audio.

In this case, the take-over request start timing calculator 51calculates the timing to start presenting a take-over request for thedriver in a normal condition, the timing being a time point which is apredetermined takeover period ahead of the estimated time at which thedriving mode switches to manual driving. For example, the takeoverperiod is estimated for each activity as a period of time necessary forthe driver to return to manual driving. With reference to FIGS. 12 and13, a description is given of how the take-over request start timingcalculator 51 calculates the timing to start presenting a take-overrequest.

The graph in FIG. 12 depicts the relation between the level ofconcentration and the timing to start presenting a take-over request ina case where the levels of wakefulness, fatigue, and nervous are ranked“medium”. L1 and L2 are threshold levels used in determining the rank ofthe level of concentration (“low”, “medium”, or “high”). In the graph,t0 indicates the timing to start presenting a take-over request to beused if the driver is in the normal condition. The graph in FIG. 13shows the relation between the level of nervousness and the timing tostart presenting a take-over request in a case where the levels ofwakefulness, fatigue, and concentration are ranked “medium”. L11 and L12are threshold levels used in determining the rank of the level ofnervousness out of the three ranks.

The storage 55 of the take-over request method determination unit 5 hasstored therein information D31 in FIG. 12 and information D32 in FIG. 13for each of the ranks of driver condition levels: “low”, “medium”, and“high”.

When the levels of wakefulness, fatigue, and nervousness are ranked“medium”, the take-over request start timing calculator 51 refers to theinformation D31, like the graph in FIG. 12, and calculates the starttiming suitable for the estimated level of concentration detected by thedriver conditions detection section 4. Thus, when the level ofconcentration is ranked “high”, the higher the level of concentration,the earlier the start timing is than the normal timing t0. Consequently,the more the driver is concentrated on their current activity, such asplaying a video game, the earlier the take-over request is presented,helping the driver to return to manual driving safely.

It is also likely that the driver needs a longer takeover time if theyare distracted. Thus, in the example shown in FIG. 12, for the level ofconcentration ranked “low”, the lower the level of concentration, theearlier the timing to start presenting a take-over request.

When the levels of wakefulness, fatigue, and concentration are ranked“medium” and the level of nervousness is ranked “high”, the take-overrequest start timing calculator 51 refers to the information D32, likethe graph in FIG. 13, and calculates the start timing suitable for theestimated level of nervousness. Consequently, the more nervous thedriver is, the earlier the take-over request is presented, helping thedriver to return to manual driving safely.

Referring back to FIG. 11, when the wakefulness level is ranked “mediumto high”, the fatigue level “low to high”, the nervousness level “middleto high”, and the concentration level “high”, the take-over requestpresentation content selector 53 sets the presentation content of thetake-over request to “give a warning prompting the driver to stop theactivity”. The take-over request presentation device selector 52determines how to present the take-over request having the warningcontent, such as displaying the notice on a particular screen of adisplay system and outputting warning audio from a speaker system, asshown in FIG. 11. To warn the driver more effectively, the take-overrequest presentation device selector 52 determines to vibrate the seatto shake the body of the driver using an actuator system as apresentation device.

If the driver is concentrated on an activity such as texting, using SNS,browsing the web, playing a video game, or watching TV, it is likelythat the driver's eyes are fixed on the screen that the driver iswatching. In view of this, based on the activity by the driver detected,the take-over request presentation device selector 52 selects the devicewhose screen is being watched by the driver (e.g., the smartphone usedfor texting or the in-vehicle TV used for watching TV) as a displaysystem used as the presentation device, as shown in FIG. 11. If theactivity in which the driver is engaging is reading, the screen of theautomotive navigation system 61 a is selected as a display system usedas the presentation device in this embodiment, as shown in FIG. 11.

When the wakefulness level is ranked “low”, the fatigue level “low tohigh”, the nervousness level “low to medium”, and the concentrationlevel “low”, the take-over request presentation content selector 53 setsthe presentation content of the take-over request to “give a warningprompting the driver to wake up”. The take-over request presentationdevice selector 52 determines how to present the take-over requesthaving such a content, such as displaying the notice on a particularscreen of a display system, outputting warning audio or an alarm from aspeaker system, vibrating the seat using an actuator system, and/orejecting cold air from the air conditioner, as shown in FIG. 11. In theabove case, the take-over request start timing calculator 51 calculatesthe start timing for the take-over request based for example on theestimated level of wakefulness, so that the take-over request will bepresented earlier than the normal timing.

3. Advantageous Effects

As described above, the driving assistance apparatus 1 according to thisembodiment assists a transition from the autonomous driving mode inwhich the vehicle is driven under autonomous control to the manualdriving mode in which the vehicle is driven by the driver. The drivingassistance apparatus 1 includes the activity detection section 3, thedriver conditions detection section 4, and the take-over request methoddetermination unit 5. The activity detection section 3 detects anactivity in which the driver is engaging while the vehicle is driving inthe autonomous driving mode. The driver conditions detection section 4detects a plurality of conditions of the driver while the vehicle isdriving. The take-over request method determination unit 5 determineshow to present a take-over request in the vehicle to inform the driverthat the autonomous driving mode is going to be cancelled. Specifically,before the autonomous driving mode is cancelled, the take-over requestmethod determination unit 5 sets a take-over request method, which is amethod of presenting a take-over request, in the presentation controller6 based on the driver's activity detected by the activity detectionsection 3 and on the driver's conditions detected by the driverconditions detection section 4.

This driving assistance apparatus 1 sets the take-over request methodbased on an activity by the driver and conditions of the driverperforming that activity. This helps the driver return to manual drivingin a transition from the autonomous driving mode to the manual drivingmode.

For example, the device to use for the presentation and the timing tostart the presentation can be changed appropriately depending not onlyon the driver's activity, but also on the physiological andpsychological conditions of the driver engaging in that activity. Forinstance, when the driver is extremely concentrated on an activity suchas playing a video game, it is likely that it will take a longer timefor the driver to be completely ready for manual driving, than for thedriver not concentrated on the activity. With this taken intoconsideration, the driving assistance apparatus 1 is configured to beable to change the timing to present the take-over request according tothe driver's level of concentration on playing the video game, which isdetected by the driver conditions detection section 4, and thereforeallows the driver to be completely ready for manual driving.

In this embodiment, the take-over request method determination unit 5(the take-over request method selection section 50) changes thetake-over request method based on the detection result of a certaincondition of the driver engaging in the detected activity (e.g., whenthe level of concentration is high), and sets the thus-changed take-overrequest method in the presentation controller 6. This enables the driverengaging in a driving-unrelated activity to return to manual drivingsafely and smoothly.

In this embodiment, the driver conditions detection section 4 detectsthe levels of a plurality of measures indicative of conditions of thedriver. The take-over request method determination unit 5 sets thetake-over request method based on the detected levels of a certaincombination of measures among the plurality of measures which issuitable for the driver's detected activity (see FIG. 10). In this way,a take-over request method is appropriately set based on the levels ofmeasures, which are one of various combinations of measures which issuitable for the activity in which the driver is engaging.

In addition, in this embodiment, the plurality of conditions include apsychological condition indicated by the level of concentration of thedriver. This enables the take-over request method to be set with thedriver's level of concentration taken into account.

In this embodiment, the driver conditions detection section 4 detectsthe level of concentration of the driver. When the driver's activitydetected is a particular activity, the take-over request methoddetermination unit 5 sets the take-over request method so that thepresentation content of the take-over request may be highlightedaccording to the detected level of concentration. This way, thepresentation content of the take-over request is highlighted more whenthe driver's level of concentration on the particular activity ishigher. This helps the driver to return to manual driving.

In this embodiment, the take-over request method determination unit 5sets the take-over request method so that the timing to start presentingthe take-over request will be advanced according to the detected levelof concentration. This way, the presentation of the take-over requeststarts earlier when the driver is over-concentrated on a particularactivity. This helps the driver to return to manual driving even more.

In this embodiment, the plurality of conditions include a physiologicalcondition of the driver. Thus, the take-over request method can be setwith a physiological condition of the driver, such as drowsiness, takeninto account.

In this embodiment, the setting the take-over request method includessetting at least one of the timing to start presenting the take-overrequest, the content presented in the take-over request, and the deviceto use to present the take-over request. This way, the take-over requestmethod can be set from various perspectives.

In this embodiment, the driving assistance apparatus 1 further includesthe first sensor group 2 and the vehicle-mounted devices IF 20 as asensor unit to sense information on the driver while the vehicle isdriving. The activity detection section 3 detects the activity by thedriver based on the sensed information. The driver conditions detectionsection 4 detects the plurality of conditions based on the sensedinformation. Such monitoring of the driver using the sensor unit enablesaccurate detection of the driver's activity and conditions.

In this embodiment, the sensor unit of the driving assistance apparatus1 includes at least one of the in-vehicle camera 21 that captures animage of the inside of the vehicle, the microphone 22 that picks upvoice in the vehicle, the vehicle-mounted devices IF 20 that receivesinformation from devices in the vehicle, the heartbeat sensor 25, theblood pressure sensor 24, and the body movement sensor 23. These varioussensor devices allow monitoring of the driver.

The sensor unit of the driving assistance apparatus 1 may include anycombination of the above sensor devices, or other sensor devices such asa perspiration sensor. The activity detection section 3 and the driverconditions detection section 4 may detect the driver's activity andconditions using information sensed by any device included in the sensorunit.

In this embodiment, the driving assistance apparatus 1 further includesthe driving information acquisition part 81 and the autonomous drivingcontinuance determination part 83. The driving information acquisitionpart 81 acquires driving information, which indicates driving conditionsof the vehicle. Based on the driving information, the autonomous drivingcontinuance determination part 83 determines, in the autonomous drivingmode, whether the autonomous driving mode will be continuable for apredetermined period of time or longer. When the autonomous drivingcontinuance determination part 83 determines that the autonomous drivingmode will not be continuable for the predetermined period of time orlonger, the take-over request method determination unit 5 sets thetake-over request method. This way, when it is determined based on thedriving information that the autonomous driving mode will not becontinuable for the predetermined period of time or longer, thetake-over request method can be set ahead of time before the autonomousdriving mode is actually cancelled.

In this embodiment, the driving information includes information on thevehicle speed, information on the vehicle position, and surroundingenvironment information, which indicates the environment surrounding thevehicle. The take-over request method determination unit 5 can thusdetermine the continuance of the autonomous driving mode based onvarious pieces of information.

In this embodiment, the surrounding environment information includesinformation on the situation of any other vehicle around the vehicle,which allows the determination on the continuance of the autonomousdriving mode to be made according to the situation of the surroundingvehicles.

In this embodiment, the vehicle is equipped with the camera 71 thatcaptures an image of the outside of the vehicle. The driving informationacquisition part 81 acquires surrounding environment information fromthe camera 71, which allows the determination on the continuance of theautonomous driving mode to be made according to any other vehicle in theimages captured by the camera 71.

In a driving assistance method according to this embodiment, the drivingassistance apparatus 1 assists a transition from the autonomous drivingmode in which the vehicle is driven under autonomous control to themanual driving mode in which the vehicle is driven by the driver. Themethod includes detecting, by the driving assistance apparatus 1, anactivity in which the driver is engaging while the vehicle is driving.The method includes detecting, by the driving assistance apparatus 1, aplurality of conditions of the driver while the vehicle is driving. Themethod includes determining, by the driving assistance apparatus 1, atake-over request method, which is a method of presenting information inthe vehicle to inform the driver that the autonomous driving mode isgoing to be canceled. The determining a take-over request methodincludes setting the take-over request method based on the detectedactivity and conditions of the driver before the autonomous driving modeis cancelled.

This driving assistance method helps the driver return to manual drivingin a transition from the autonomous driving mode to the manual drivingmode.

Other Embodiments

The comprehensive or specific aspects described above may be implementedas one or any combination of a system, a method, an integrated circuit,a computer program, or a computer-readable recording medium, such as aCD-ROM.

The present disclosure includes the various methods described above. Inone aspect of the present disclosure, these methods may be a computerprogram executed by a computer, or digital signals forming the computerprogram.

In one aspect of the present disclosure, the computer programs or thedigital signals may be recorded in a computer-readable recording medium,such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM,a DVD-RAM, a Blu-ray Disc (BD, registered trademark), a USB memorydevice, a memory card, such as an SD card, or a semiconductor memory. Anaspect of the present disclosure may be the digital signals recorded inany of these recording media.

In one aspect of the present disclosure, the computer program or thedigital signals may be transmitted using a telecommunications line, awireless or wired communications line, a network such as the Internet,data broadcasting, or the like.

One aspect of the present disclosure may be a computer system includinga microprocessor and a memory, the memory having the computer programstored therein, the microprocessor being operated according to thecomputer program.

The above-described apparatus may be implemented by an independentcomputer system by transfer of the program or the digital signalsrecorded in the recording medium or by transfer of the program or thedigital signals over the network.

The numerical values used in the above description have been provided togive a specific illustration of the present disclosure, and the presentdisclosure is not limited to such numerical values given forillustrative purposes.

The functional blocks in the block diagrams are just an example in termsof how they are divided. Two or more functional blocks may beimplemented as one functional block, or one functional block may bedivided into two or more functional blocks. One or more functions of acertain functional block may be implemented by another functional block.Similar functions of different functional blocks may be processed inparallel or by time division by single hardware or software.

The order in which the steps of the driving assistance method areexecuted has been specified to provide a specific illustration of thepresent disclosure. The steps may be executed in any other order. Inaddition, one of the steps may be executed at the same time as (inparallel with) another step.

The driving assistance apparatus according to one or more aspects hasbeen described above based on the embodiment, but the present disclosureis not limited to the embodiment. The one or more aspects also includevarious modifications of the embodiment that may be conceived of bythose skilled in the art and modes formed in combination with aconstituent of a different embodiment, as long as such modifications ormodes do not depart from the gist of the present disclosure.

The driving assistance apparatus and the driving assistance methodaccording to the present disclosure are applicable to an autonomousdriving system that automates driving of the vehicle.

What is claimed is:
 1. A driving assistance apparatus for a vehicle,comprising: one or more memories; and a circuitry that performs, whenthe vehicle is traveling under an autonomous driving mode, operationsincluding: detecting, using at least a sensor, an activity in which adriver is engaging, and a psychological state that reflects a degree ofconcentration of the driver on the activity, determining, based ondriving information indicating driving conditions of the vehicle,whether the autonomous driving mode is continuable for a period of time,and when it is determined that the autonomous driving mode is notcontinuable for the period of time, determining, based on at least theactivity and the psychological state, a method of presenting, in thevehicle, a take-over request that informs the driver that the autonomousdriving mode is going to be cancelled.
 2. The driving assistanceapparatus according to claim 1, wherein the determining of the method ofthe presenting includes determining, according to the degree of theconcentration, a degree to which the take-over request is highlighted.3. The driving assistance apparatus according to claim 1, wherein thedetermining of the method of the presenting includes determining,according to the degree of the concentration, a timing to start thepresenting of the take-over request.
 4. The driving assistance apparatusaccording to claim 1, wherein the operations further includes detectinga physiological state of the driver, and the determining of the methodof the presenting is performed based on at least the activity, thepsychological state, and the physiological state.
 5. The drivingassistance apparatus according to claim 1, wherein the determining ofthe method of the presenting includes determining at least one of (i) atiming to start the presenting of the take-over request, (ii) content tobe presented as the take-over request, or (iii) a device by which thetake-over request is presented.
 6. The driving assistance apparatusaccording to claim 1, wherein the at least a sensor includes at leastone of a camera, a microphone, an in-vehicle interface, a heartbeatsensor, a blood pressure sensor, or a body movement sensor.
 7. Thedriving assistance apparatus according to claim 1, wherein the drivinginformation includes information indicating at least one of a speed ofthe vehicle, a position of the vehicle, or an environment around thevehicle.
 8. The driving assistance apparatus according to claim 7,wherein the information of the environment around the vehicle includesinformation indicating other vehicle existing around the vehicle.
 9. Thedriving assistance apparatus according to claim 1, wherein the activityis an activity unrelated to driving of the vehicle.
 10. A drivingassistance apparatus for a vehicle, comprising: one or more memories;and a circuitry that performs, when the vehicle is traveling under anautonomous driving mode, operations including: detecting, using at leasta sensor, an activity in which a driver is engaging, and a psychologicalstate that reflects a degree of concentration of the driver on theactivity, and determining, based on at least the activity and thepsychological state, a method of presenting, in the vehicle, a take-overrequest that informs the driver that the autonomous driving mode isgoing to be cancelled, wherein when the driver is watching a screen of adisplay in the activity, the takeover request is present on the screen.11. The driving assistance apparatus according to claim 10, wherein thedisplay is included in a smartphone, an automotive navigation system ora TV system.